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In this study, we asked what effect survey length has 
on student non-response rates to individual items on 
IDEA’s Diagnostic Feedback (DF) and Learning 
Essentials (LE) forms. Because the DF has more items 
than the LE (41 vs. 19), users sometimes ask whether 
the non-response rates to individual items are higher 
on the DF. Item non-response can be a concern when a 
substantial proportion of students within a class do not 
respond to all of the survey questions, and those 
students have characteristics—related to what is being 
measured—that differ from respondents who 
completed all items on the survey (Dillman, 2007). 


Our approach was to analyze individual student ratings 
of classes contained in the 2015-2016 IDEA-CL 
database. Specifically, we computed item non- 
response rates in individual ratings of 298,209 
students enrolled in 36,629 classes who used the DF, 
and 39,899 enrolled in 5,120 classes who responded 
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to the LE. Tables 1 and 2 show item response rates on 
the 41-item DF and the 19-item LE. The last item on 
each survey is an open-ended question for students to 
write comments. 


The DF asks students how frequently the instructor 
used each of 19 teaching methods and how much 
progress they made on 13 learning objectives. 
Students also are invited to respond to six items about 
themselves and the course, two single-item summary 
questions, and the open-ended question for written 
comments. As shown in Table 1, the response rates to 
the first 19 questions were all 100%. For the 13 
learning objectives, it dropped slightly to 99.2%, and 
for the six student and course characteristics it was 
99.0%. So, approximately 99% of the students who 
responded to the DF answered all 40 forced-choice 
questions. A little over half (52.9%) offered written 
comments. 


Item Response Rates on the Diagnostic Feedback Form (N = 298,209) 


Items 


% 


Teaching methods (Items 1 - 19) 


Progress on learning objectives (Items 20 - 32) 
Student and course characteristics (Items 33 - 38) 


Overall summary measures (Items 39 - 40) 


Comments (Item 41) 


On the LE, students rate the progress they made on 13 
learning objectives, respond to three questions about 
themselves, provide judgments on two overall summary 
measures, and offer written feedback if they desire. 
The non-response pattern for the LE was very similar to 
the DF (see Table 2). Of those who responded to the 
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survey, 100% answered all 13 learning objectives, and 
99.2% responded to the three questions about 
themselves and the two summary measures. Again, 
slightly more than half (53.5%) provided written 
comments. 
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Table 2 
Item Response Rates on Learning Essentials Form (N = 39,899) 

Items % 
Progress on learning objectives (Items 1 - 13) 100.0 
Student characteristics (Items 14 - 16) 99.2 
Overall summary measures (Items 17 - 18) 99.2 
Comments (Item 19) 53.5 
To summarize, about 99% of students respond to all References 


forced-choice questions regardless of whether 
instructors use the DF or LE. Non-response bias most 
likely, then, imposes very little threat to student 
responses on individual items, and should not affect 
overall teaching evaluations. We can most likely 
conclude the missing responses occur when students 
elect not to advance to the next page in the survey. 
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